WIMP - Is This 3 Sigma Signal The Road To Dark Matter?
5-sigma is a modern designation for ‘discovery’ but 3-sigma or even 5-sigma is not telling the whole story. A 3-sigma result means there is a 1 in 370 chance it could just be wrong - polls are 2-sigma and that 95% confidence interval is why individually they don’t have much value but, as we saw in the last US presidential election, averaging a bunch of polls makes for a lot better result - every statistician who made a call the day before election got every state right just by averaging a dozen polls. 5-sigma is something else. At 5-sigma the odds of a crazy result are down to 1 in over 1.7 million. The top quark discovery by the Tevatron at Fermi National Accelerator Laboratory (FNAL) in the 1990s, and the collaboration referring to it as ‘evidence’ before the 5-sigma ‘discovery’ result, is the time when physics commonly established 5-sigma as the discovery threshold.
But 5-sigma is not some magic number, it is still just a statistic. Sometimes people don’t know what they are doing, or they want to manipulate results for legitimacy, so you instead have to understand how consistent their data is with the null hypothesis, a much trickier proposition. Fortunately, particle physics is filled with cranky people who like to argue and show each other up so errors are not likely to happen and deception is impossible. The benefit for us all is that the field uses p-values correctly as a result.
So a portion of the quote above should read, “every statistician AND MANY PEOPLE, EXCEPT MITT ROMNEY, PAUL RYAN, THIER ENTIRE STAFFS, ALL OF FAUX NEWS INCLUDING CARL ROVE, who made a call the day before election got every state right just by averaging a dozen polls.”